Using the Visual Denotations of Image Captions for Semantic Inference

Semantic inference is essential to natural language understanding. There are two different traditional approaches to semantic inference. The logic-based approach translates utterances into a formal meaning representation that is amenable to logical proofs. The vector-based approach maps words to vectors that are based on the contexts in which the words appear in utterances. Real-valued similarities are used in place of logical inferences. We introduce the notion of the visual denotation of an utterance, which is the set of images that it describes. This notion borrows the abstract concept of a denotation of an utterance as the set of possible worlds in which the utterance is true from the logic-based approach, and instantiates possible worlds as images. In this dissertation, we also show how visual denotations can be created for descriptions of everyday entities and events. Additionally, we demonstrate that visual denotations can be used as a new model of semantic similarity, and that this model is better at identifying entailment relations between descriptions of images than traditional distributional similarities. In order to do this, we create an image caption corpus consisting of captions and images depicting everyday actions. This corpus has a number of useful features that would assist in investigating everyday events and the different ways in which they can be described. We use the captions in the corpus as the starting point for producing caption fragments with larger visual denotations. We accomplish that by creating a denotation graph, a subsumption hierarchy over the captions that links captions and the images that depict them, that also allows for the visualization and navigation of the image caption corpus in an intuitive manner.